Statistically comparing EEG/MEG waveforms through successive significant univariate tests: how bad can it be?

When making statistical comparisons, the temporal dimension of the EEG signal introduces problems. Guthrie and Buchwald (1991) proposed a formally correct statistical approach that deals with these problems: comparing waveforms by counting the number of successive significant univariate tests and then contrasting this number to a well-chosen critical value. However, in the literature, this method is often used inappropriately. Using real EEG data and Monte Carlo simulations, we examined the problems associated with the incorrect use of this approach under circumstances often encountered in the literature. Our results show inflated false-positive or false-negative rates depending on parameters of the data, including filtering. Our findings suggest that most applications of this method result in an inappropriate familywise error rate control. Solutions and alternative methods are discussed.